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Chapter I - Introduction and Background 



Introduction 



The Legislative Audit Committee (LAC) designated a review of 
accessibility to management information from the Child and Adult 
Protective Services (CAPS) system as a "high priority" for 
performance audit work in the 2005 biennium. The performance 
audit objectives addressed the following CAPS concerns: 



► Ability and time needed to generate reports and statistics. 

► Accuracy and completeness of reporting. 

► Difficulties using the system. 



Survey Scope And 
Methodologies 



Our initial audit work focused on, "What steps have been taken by 
the Department of Public Health and Human Services (DPHHS) to 
improve accessibility to management information from CAPS?" 

While several agencies and numerous personnel use CAPS, our 
review concentrated on the Child and Family Services Division 
(CFSD), which is the primary data collection entity and user of 
CAPS information. In addition to CFSD, we examined the role of 
the department's Operations and Technology Division (OTD), which 
is responsible for CAPS programming and maintenance. 



We interviewed department personnel regarding the CAPS system 
and types of management information generated. We obtained and 
reviewed various documents including: 

► Federal and state regulations related to the CAPS system. 

► Reports regarding federal reviews of CAPS and similar state 
systems. 

► CFSD management team meeting minutes. 

► CAPS contract information and reports. 

► CAPS system documentation related to the original development 
effort. 

► The most recent information technology plan for the department. 

► A listing of standard CAPS reports. 

► Examples of other management information reports. 
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One of our main areas of focus was review of the system used to 
document and track requests for system maintenance and 
modification, known as Tracker. We analyzed the data in the 
Tracker system regarding all CAPS system requests. We observed a 
CAPS change control meeting to help understand how priorities are 
established for CAPS change requests. We also examined the 
process used to develop CAPS software once a change request is 
approved. 



Background 



We coordinated our review with two other Legislative Audit 
Division audits of DPHHS: 1) Foster Care Parent Program (04P-03), 
and 2) Workload/Caseload (04P-09). 

For many years, concerns were raised about the lack of information 
available on children in foster care and their families. To address 
some of these concerns. Congress amended Title IV -E of the Social 
Security Act in 1986 requiring the federal government to institute a 
foster care and adoption data collection system. 



The Omnibus Budget Reconciliation Act of 1993 provided federal 
funding for statewide automated child welfare information systems 
to carry out the states' programs under Title IV of the Social Security 
Act. This funding was available from October 1, 1993, through 
September 30, 1996, for the planning, design, development and 
installation of statewide systems. Montana received $4,827,896 of 
federal funding and provided $1,609,299 of state funding to create 
CAPS. 



Montana's SACWIS System 



CAPS is a statewide online information system used by DPHHS, the 
Department of Corrections, and Tribal Social Services agencies. 
CAPS supports monitoring of foster care cases, adoption cases, 
provider contracts and licensing, financial accounting, payments for 
services to providers, and reporting functions. It is required to meet 
federal Statewide Automated Child Welfare Information System 
(SACWIS) standards, and provide federal reporting requirement 
information for the Nationwide Child Abuse and Neglect Data 
System (NCANDS). 
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CAPS became operational in 1996 and was designed to provide the 
following services: 

► Automated payment approval process and warrant issuance to 
service providers. 

► Automated case management by tracking all contacts made with 
individuals associated with the case. CAPS tracks court actions, 
maintains client information such as relationships, addresses, 
aliases, educational background, medical history, special needs, 
and financial resources. 

► Maintain provider information such as licensing requirements, 
placement history for a provider facility, services provided, 
provider rates, key personnel, training, and payment history. 



CAPS System Maintenance CAPS has ongoing operational and maintenance needs. These 

system maintenance needs are met through a DPHHS contract with a 
private company. The purpose of the CAPS "facilities management" 
contract is to provide: application support, user training, help desk 
support, and production operations. A facilities management 
contract has been in place since initial system development in 1996. 
The term of the current facilities management contract is July 1 , 
1999 through June 30, 2004. The cost for system maintenance under 
this contract (5 years) was $7,808,928. Per contract clause, DPHHS 
has extended the contract for an additional year to June 30, 2005. 
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Introduction 



CAPS Change Process 



Chapter II - Accessibility to CAPS 
Management Information 

Our review concentrated on the Child and Adult Protective Services 
(CAPS) system and procedures used for generating management 
information. This chapter presents information on the process used 
to change the CAPS system, as well as statistics on the number and 
types of change requests submitted over the past several years. The 
chapter also contains information on reviews of the CAPS system 
and user difficulties. 

One of our objectives was to review the ability and time needed to 
generate reports and statistics from CAPS. The main process used 
by the department for generating non-standard management 
information reports is the CAPS change request process. This 
process includes generation of new standard management 
information reports, as well as ad hoc reports. 



CAPS is a mainframe computer system written in COBOL, a 
programming language developed in the late 1950's. The CAPS 
change process is a five-step process beginning with analysis of the 
problem or requested enhancement. The following flowchart details 
the CAPS software development process used by the Operations and 
Technology Division, Department of Public Health and Human 
Services (DPHHS). 
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Figure 1 
CAPS Software Development Process 
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e: Compiled by the Legislative Audit Division from department records. 



While this process can be time consuming, it is a standard approach 
for this type of system, and is used when implementing CAPS 
maintenance and enhancement requests, as well as generating ad hoc 
reports. 



CAPS Change Requests 



Requests for changes to CAPS originate from system users who 
report problems and request enhancements and reports. Requests for 
changes and enhancements are routed through two channels 
depending on the necessity of the request. Most requests go to the 
CAPS Change Control Committee, which determines the importance 
and level of priority for each request received. Change Control 
Committee membership includes department employees from the 
central office, as well as the field. 
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Figure 2 

CAPS Change Requests - Total 

(1995-2003) 
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Note: Youth Court Requests do not show on the figure due to the low number of requests. 
Source: Compiled by the Legislative Audit Division from department records. 



There are several types of change requests including: 

► Enhancement - add something new to the system. 

► Maintenance - keep the system operating as intended. 

► Ad Hoc - special request for changing the system. 

► Youth Court - maintenance/enhancement related to Youth Court. 

As can be seen in the figure above, maintenance requests have 
dominated system change requests for the past four years. 



Request Change List 



A request change list is maintained and reviewed by the CAPS 
Change Control Committee. Over four thousand requests have been 
registered on the list since CAPS implementation in 1996. These 
requests range from basic maintenance, to enhancements for 
improved customer service, to management information reports. 
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The Change Control Committee meets quarterly to discuss and 
prioritize requests. During the quarterly meeting, the committee is 
only able to address the top 20 or so requests due to resource 
limitations (number of contracted programmers available). 

The number of projects being worked on varies according to 
the number of available programmers (currently five) and the 
complexity of the project. As a project is completed, new 
and existing requests are reprioritized. Requests are 
categorized as a one, two, or three according to their priority 
status, with one being the highest priority. The following 
figure shows priorities for open requests as of March 2004. 



Figure 3 

CAPS Change Control Committee Open Request Priorities 

(as of March 2004) 
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Source: Compiled by the Legislative Audit Division from department records. 
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The following are examples of high priority requests: 



CAPS Open Change 
Requests 



► Maintenance (#39 1 2): Third-party cost of care journal 
not processing correctly in SABHRS. 

► Enhancement (#422): Management report listing 
children in need of permanency hearings. 

► Ad Hoc (#3976): One-time extract report for use in 
comparing CAPS and Medicaid data to identify 
duplicate payments. 

Due to the large amount of requests for enhancements and 
maintenance, there is a backlog of unaddressed requests. Since 
December 2003, open requests on the change list have remained 
fairly static due to a large number of ad hoc requests in the past. As 
of September 2004, there were 408 open requests as shown in the 
following table. 



Table 1 

CAPS Open Change Requests 

(as of September 2004) 



Type 


March 2004 


April 2004 


Sept 2004 


Maintenance 


348 


322 


285 


Enhancement 


128 


139 


123 


Ad hoc 


3 


2 





Total 


479 


463 


408 



Note: Youth Court Requests are not included 

Source: Compiled by the Legislative Audit Division from 
department records. 



Based on our preliminary review of the request list, it is difficult to 
determine which requests are federal mandates, versus state 
mandates, versus department requests. In addition, it is difficult to 
accurately determine which requests are "management information 
reporting" related. What is clear is that priorities are continually 
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Ad Hoc List 



Summary 



changing, so completion of change requests is impacted on an 
ongoing basis. 

A separate request list is maintained to keep CAPS up-to-date with 
federal and state requirements. Since federal and state mandates and 
program compliance changes are often required to be completed in a 
short amount of time, these enhancements cannot be properly 
addressed through the normal change request process. The ad hoc 
list allows the request to circumvent the process and be addressed in 
the shortest amount of time possible (i.e. -these requests take 
precedence over other requests). Most of the requests on the ad hoc 
list are generated by federal requirements. Legislative and 
departmental requests are also included on this list if they are needed 
in a short amount of time. In April 2004, there were 2 requests on 
the ad hoc list, and as of September 2004, this list was reduced to 
zero. 

The ability to create and generate reports exists, but the time 
needed can be lengthy. 



Conclusion 



CAPS Reviews 



The department has a process in place for generating standard and 
new management information reports from CAPS. However, due to 
the amount of changes requested, only some of the requests can be 
addressed right away. Based on CAPS system design, there is a 
required process that is used to change the system. 

The type of system, available resources, and process used to 
prioritize requests create circumstances that cause delays in 
completing requests in a timely manner. 

Another one of our objectives was to review the accuracy and 
completeness of CAPS reporting. The Child and Family Services 
Division (CFSD) is currently reviewing CAPS reports to try to 
improve reporting. The Division is identifying reports where the 
same numbers or counts are calculated, reviewing the calculation 
programming code, and eliminating duplicates and variations of 
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calculations. The purpose is to increase consistency /accuracy of 
reports by only using one method of calculation. 



Federal Review of CAPS xhe federal government is continually reviewing foster care and 

CAPS. One level of review relates to certification of Statewide 
Automated Child Welfare Information Systems (SACWIS). Because 
federal funding was used for system development, states must 
receive certification. Only five states have been fully certified by the 
U.S. Department of Health and Human Services (HHS). Montana is 
not one of the five. The department continues to work toward 
certification of CAPS, which involves completion of interfaces with 
other systems. 

In August of 2002, a federal review of Montana's Child and Family 
Services Program was conducted by a team of HHS reviewers who 
examined Montana's system to assure state performance in the areas 
of safety, permanency, and well being with regard to seven factors. 
Part of the review required federal reviewers to inspect Montana's 
SACWIS system, CAPS. In the final report the federal reviewers 
noted CAPS as an area of strength for Montana. It was a strength 
because the State has the automated capacity to track the required 
information on children in foster care, such as their status, 
demographic characteristics, location, and goal. 

In addition, the U.S. Government Accountability Office (GAO) 
completed a study of child welfare related to states' development of 
SACWIS systems. The following excerpt is from the final report: 

"Several factors affect states' ability to collect and report reliable 
data on children served by state child welfare agencies, and some 
problems exist, such as a lack of clear and documented guidance, 
with HHS's oversight and technical assistance. ...although 
states were mandated to begin reporting Automated Foster Care 
and Adoption Reporting System (AFCARS) in 1995, few 
reviews of states' AFCARS reporting capabilities have been 
conducted to assist states in resolving some of their challenges." 
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LAD Review 



Summary 



The Legislative Audit Division (LAD) conducted a limited scope 
Information Systems (IS) audit (03DP-05) of five DPHHS systems, 
including CAPS, in 2003. The results of the IS audit indicated 
CAPS is operating as intended. The audit tested selected edits 
including payment edits and eligibility determination, and ensured 
data validations were functioning as intended. The audit also 
reviewed the CAPS report for reconciling transactions between the 
system and SABHRS. In addition, LAD completed an IS audit of 
CAPS in 1997 (97DP-06). This audit reviewed general and 
application controls within the data processing environment. 

The accuracy and completeness of CAPS reporting is reviewed. 

Our current audit work did not test CAPS reporting accuracy and 
completeness since several entities conduct reviews in this area. In 
addition, DPHHS is conducting its own review of CAPS to try and 
improve report accuracy and consistency. 



Conclusion 



Difficulties Using CAPS 



The department and other entities are addressing report accuracy 
and completeness issues. 

Difficulty using the CAPS system has been a concern expressed by 
users on an ongoing basis and relates to our third objective. We 
believe this is more a concern with system design and the multiple 
screens used to input/extract data. CAPS was initially designed as an 
automated payment approval and case management system, not for 
individuals to generate reports and statistics. Thus, as detailed 
previously in this chapter, the process for obtaining management 
information can be lengthy. 



Individuals commenting on this issue during the federal 
government's review of the system expressed favorable opinions 
regarding CAPS. They noted the system is easily accessible through 
remote access after hours and on weekends, generates helpful reports 
used by managers and administrators, and you can access 
information on all children in foster care, including locations, goals, 
and treatment plans. According to individuals, CAPS can identify 
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the names, status, location, and goals of all children in foster care 
and the system meets the needs of staff in their day-to-day activities. 
A few individuals suggested the system is time-consuming and not 
user friendly. One concern noted was there are too many screens and 
they are difficult to navigate if the user does not have significant 
computer knowledge. 



Summary 



Individuals also noted all CFSD staff receive a week of training on 
CAPS and report the training is helpful. The department has an 
ongoing training component for CAPS. Training is conducted for 
new employees, as well as providing updates for existing employees. 
While training provides technical instruction for use of CAPS, it 
cannot provide complete instruction on how to address every 
situation experienced during use of CAPS. The number of screens 
(online forms) in CAPS is in the hundreds, so it can be 
overwhelming to some users. Changes and enhancements to CAPS 
have added to the number of screens and changed procedures for 
using the system. These changes impact CAPS users and can create 
difficulties with using the system. 

Some users experience difficulties using CAPS. 



Conclusion 



Future Plans 



Some users experience difficulties using CAPS. These difficulties 
can relate to the extensive number of screens used to input/extract 
data, which is a result of system design and ongoing system 
enhancements. The department has an ongoing CAPS training 
component to assist users and address difficulties with system use. 

Difficulties with CAPS use relate to the extensive number of 
screens and original system design, which are partially addressed 
through available training. 

Although significant enhancements were made to CAPS, the 
department believes it has outlived its expected eight year lifespan. 
The DPHHS Information Technology (IT) Plan includes a 
recommendation to replace CAPS with a new system in fiscal year 
2006. Based on its review, the CAPS IT Planning Committee 
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developed the following recommendations for the CAPS successor 
system: 

► CAPS should be capable of incremental growth, automatic 
collection and exchange of information across an adverse set of 
systems and service providers, and have the ability to develop 
reports and query data that is electronically submitted to state 
and federal entities. 

► CAPS will support the Common Client Database concept with 
the overall vision of improving outcomes for Montana's 
vulnerable citizens. 

► CAPS users will have the ability to extract data from a data 
warehouse to create their own reports for legislative purposes, 
state advisory councils, local community involvement, 
management analysis, etc. 

► CAPS users will use a common client identifier and have the 
ability to interface with the Social Security Administration for 
SSN verification. 

► CAPS will have enhanced interfaces with TEAMS (The 
Economic Assistance Management System), SEARCHS (System 
for the Enforcement and Recovery of Child Support), CCUBS 
(Child Care Under the Big Sky), etc. 

► CAPS will provide online access to criminal background 
information. 

► CAPS will have the ability to interface with the appropriate 
jurisdiction to perform criminal background checks, using the 
best technology available. 

If the department's future plans are realized and a successor system 
is implemented, the concern of difficulties using the system may or 
may not be reduced. Users must learn how to use a new system, so 
difficulties may be increased. As time passes and users become 
comfortable using a new system, difficulties, at least for some users, 
may be reduced. 



Overall Conclusion An Audit Does Not Appear Necessary. 



The objectives of our review were related to CAPS concerns and 
included: 

► Ability and time needed to generate reports and statistics. 
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► Accuracy and completeness of reporting. 

► Difficulties using the system. 

Our review indicates CAPS has the ability to generate reports and 
statistics, but the time needed can be lengthy. The accuracy and 
completeness of CAPS reporting is reviewed by both federal and 
state entities. Finally, CAPS users may experience difficulties using 
the system, which is addressed through the department's ongoing 
training component and future plans. 

The answer to our audit planning question, "What steps have been 
taken by the department to improve accessibility to management 
information from CAPS?" is answered through maintenance and 
enhancements to the system. The information provided to the federal 
government is used to manage the program. While Montana did not 
receive a perfect score on its last review, for the most part, CAPS is 
meeting federal government requirements. In relation to 
management information reporting requests from other sources, 
namely the legislature and department, the department addresses 
these requests in the same manner. CAPS does not have the ability 
to adjust quickly to change or the capacity to create user-generated 
reports from a data warehouse. Future department plans address 
these issues. 

Due to the age and design of the CAPS system, and the department' s 
recommendation for a successor system, more audit work does not 
appear necessary. Based on our review, it appears the two main 
issues a future system needs to address, in relation to management 
information reporting, are the ability to adjust to change and the 
capability for user-generated reports. Based on department IT Plan 
recommendations, these two areas are being considered, and future 
audit work on CAPS reporting is not necessary at this time. If a new 
system is approved in the future, further audit involvement could be 
re-evaluated at that point in time. 
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